Pioneers of Parametrics

This paper provides a historical account of the development of the field of parametrics through information obtained during interviews of twelve pioneers of the field. Cost model developers, users, and practitioners were interviewed with the intent to capture their views on the impact between cost estimation research and practice. The individuals interviewed represent a diverse range of perspectives including academia, government, and industry. Each perspective sheds light on the areas in which the field of parametrics has had an impact and which synergies have been influential in the development of the field. The implications of the findings are discussed in light of the future challenges for the field of parametrics

Systems Engineering Cost Estimation Across BAE Systems: Trans-Atlantic Collaboration and Identification of Future Opportunities

As organizations develop more complex systems, increased emphasis is being placed on Systems Engineering (SE) to ensure that cost, schedule, and performance targets are met. Correspondingly, the failure to adequately plan and fund the systems engineering effort appears to have contributed to a number of cost overruns and schedule slips, especially in the development of complex aerospace systems. This has resulted in a recent increased emphasis on revitalizing systems engineering in government and commercial organizations. COSYSMO, the Constructive Systems Engineering Cost Model, is an “open” model that can help people reason about their decisions related to systems engineering through a structured approach for estimating systems engineering effort. BAE Systems, in close collaboration with University of Southern California (USC) Center for Systems and Software Engineering (CSSE) and the MIT/Lean Aerospace Initiative (LAI), has been intimately involved in the development and validation of the model since its inception and continues to collaborate on the refinement of the model

Zen in the Art of Cost Estimation

Engineering cannot wait until all phenomena are explained. Engineers may work effectively, often for centuries, with heuristics. This paper provides over thirty heuristics that have been inspired by the development and application of a systems engineering cost estimation model. The objective of this paper is to present such heuristics in a simple manner so that they can benefit those that develop, calibrate, and use cost models

A Theory of Objective Sizing

The process of building parametric models to estimate the cost of large scale complex systems have recently uncovered unanticipated challenges. The most difficult of which includes the ability to define the boundary of the system being estimated. This boundary is an essential step towards determining the size of the system; a major input into parametric models. In this paper, we build on a concept from psychology known as the moon illusion to develop a theory of objective sizing. This theory has two main benefits: it helps explain why stakeholders have different views of systems and it provides an approach for how these differences can be reconciled. Ultimately it will help technical communities arrive at a more objective way for measuring system size which will ultimately improve the accuracy and influence of parametric models

Optimizing Optimism in Systems Engineers

Biases continue to be an important aspect of human judgment and decision making because they often occur subconsciously and can frequently lead to unfavorable outcomes. Optimism bias is one type of cognitive illusion that is often overlooked because of its association with good health and positive outcomes. However, the existence of optimism bias in human judgment can be very damaging especially when it distorts a person's view of future events. In order to better understand optimism bias we explore the benefits and downsides of optimism as well as some empirically-based origins of both optimism and pessimism. This provides a backdrop for a methodology for quantifying optimism and pessimism followed by a discussion about certain professions that make well-calibrated decisions. Results are explored from an optimism survey given to a cohort of eighty systems engineers, which ultimately portray the degree to which optimism bias influences decision making in the estimation of cost and schedule of large projects. A calibration exercise is designed to calibrate optimism in systems engineers with the ultimate goal of helping cost estimation realism. Finally, prescriptive advice is provided to help individual decision makers better optimize their optimism. The implications of this work are twofold. First, the mechanism for quantifying optimism in systems engineers provides useful insight into the degree of optimism that exists among this group of decision makers. This can influence a number of decision making processes that may traditionally be seen as immune from biases due to their routine nature. Second, the process for calibrating optimism provides a way to validate the effectiveness of optimism reduction techniques on different types of decision makers. It also helps to distinguish between certain people who are more receptive to bias corrections and are therefore more likely to be better estimators in real life

Myth Buster: Do Engineers Trust Parametric Models Over Their Own Intuition?

This paper explores the abilities of engineers to estimate everyday tasks and their reliance on their own intuition when performing cost estimates. The approach to answering these questions is similar to that of the popular television show MythBusters which aims to separate truth from urban legend using controlled experiments. In MythBusters, methods for testing myths and urban legends are usually planned and executed in a manner to produce the most visually dramatic results possible, which generally involves explosions, fires, or vehicle crashes. While the question of parametric models versus intuition is not as exciting, we provide an interesting result that demonstrates the difference between what is real and what is fiction in the world of cost estimation. Two heuristics, representativeness and anchoring, are explored in two experiments involving psychology students, engineering students, and engineering practitioners. The first experiment, designed to determine if there is a difference in estimating ability in everyday quantities, demonstrates that the three groups estimate with relatively equal accuracy. The results shed light on the distribution of estimates and the process of subjective judgment. The second experiment, designed to explore abilities for estimating the cost of software-intensive systems given incomplete information, shows that predictions by engineering students and practitioners are within 3-12% of each other. Results also show that engineers rely more on their intuition than on parametric models to make decisions. The value of this work is in helping better understand how software engineers make decisions based on limited information. Implications for the development of software cost estimation models are discussed in light of the findings from the two experiments

Sea Level Requirements as Systems Engineering Size Metrics

The Constructive Systems Engineering Cost Model (COSYSMO) represents a collaborative effort between industry, government, and academia to develop a general model to estimate systems engineering effort. The model development process has benefited from a diverse group of stakeholders that have contributed their domain expertise and historical project data for the purpose of developing an industry calibration. But the use of multiple stakeholders having diverse perspectives has introduced challenges for the developers of COSYSMO. Among these challenges is ensuring that people have a consistent interpretation of the model’s inputs. A consistent understanding of the inputs enables maximum benefits for its users and contributes to the model’s predictive accuracy. The main premise of this paper is that the reliability of these inputs can be significantly improved with the aide of a sizing framework similar to one developed for writing software use cases. The focus of this paper is the first of four COSYSMO size drivers, # of Systems Requirements, for which counting rules are provided. In addition, two different experiments that used requirements as metrics are compared to illustrate the benefits introduced by counting rules

Navigating the Metrics Landscape: An Introductory Literature Guide to Metric Selection, Implementation, & Decision Making

The focus of this paper is to depict the vast landscape of literature related to enterprise performance measurement in a concise and comprehensible manner for researchers and practitioners. We focus particularly on the enterprise as the unit of analysis and consider measurement systems from stakeholders at all levels. A broad range of considerations will be explored, ranging from micro-level considerations such as employee performance measurement to macro-level considerations such as enterprise measurement systems. Moreover, we discuss measurement-related problems identified in practice and solutions proposed in academic literature. To illustrate this evolution of measurement knowledge over time, we discuss the effects of metrics from three distinct viewpoints: (1) selecting the right metrics, (2) creating and implementing measurement frameworks; and (3) metrics for decision making

On the Use of Architectural Products for Cost Estimation

The Department of Defense Architecture Framework (DoDAF) provides a standard set of views that illustrate specific attributes of a system. These views give different levels of detail and purpose that allow engineers to express operational, system, technical, and architectural properties for specific purposes. The twenty six different views available can be useful and at the same time overwhelming to someone unfamiliar with the framework. An increasing number of defense contractors are using DoDAF to characterize system attributes. These same contractors are responsible for providing cost estimates for the development and implementation of systems. This paper provides the link between these two areas by relating architectural views to system representation for cost estimation. There are several benefits to this link. First, the cost estimation community can benefit from a deeper understanding of the DoDAF and its objectives to improve the field of cost estimation through the development of models that better represent system architectures. Second, DoDAF can serve as a common language between customers and contractors by improving the representation of stakeholder needs and objectives. Third, the architecting community can benefit from the identification of subjective cost drivers currently not addressed in the DoDAF products. In this spirit, this paper describes how DoDAF architecture frameworks can be used to determine functional system size for adequate estimating of systems engineering effort. This is illustrated through the use of the OilCo FastPass system defined in previous work. The utility of using the FastPass system is that it is well documented in journal articles and it is a system familiar to the general systems engineering audience